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Abstract-  The  purpose  of  this  paper  is  to  evaluate 
the  physical  and  mental  stress  based  on  the  physio¬ 
logical  index,  and  a  new  evaluation  method  of  heart 
rate  variability  is  proposed.  This  method  combines 
the  wavelet  transform  with  a  recurrent  neural  net¬ 
work.  The  features  of  the  proposed  method  are  as 
follows:  1.  The  wavelet  transform  is  utilized  for 
the  feature  extraction  so  that  the  local  change  of 
heart  rate  variability  in  the  time-frequency  domain 
can  be  extracted.  2.  In  order  to  learn  and  evalu¬ 
ate  the  different  patterns  of  heart  rate  variability 
caused  by  individual  variations,  body  conditions, 
circadian  rhythms  and  so  on,  a  new  recurrent  neu¬ 
ral  network  which  incorporates  a  hidden  Markov 
Model  is  used.  In  the  experiments,  a  mental  work¬ 
load  was  given  to  five  subjects,  and  the  subjective 
rating  scores  of  their  mental  stress  were  evaluated 
using  heart  rate  variability.  It  was  confirmed  from 
the  experiments  that  the  proposed  method  could 
achieve  high  learning/evaluating  performances. 
Keywords-  Heart  rate  variability,  Mental  stress, 
Wavelet  transform,  Recurrent  neural  network, 
Hidden  Markov  model. 

I  Introduction 

An  electrocardiogram  (ECG)  is  available  for  a  basic 
physiological  index  which  evaluates  the  change  in  patient’s 
body  condition  and  monitors  physical  and  mental  stress  in 
his/her  daily  activities.  The  heart  rate  is  complicated  since 
it  is  affected  by  various  factors  such  as  a  sinus  node  for  a 
pacemaker,  autonomic  nerves,  the  endocrine  system  and  so 
on.  It  is  expected  that  the  changes  of  these  factors  can  be 
evaluated  based  on  the  analysis  of  Heart  Rate  Variability 
(HRV).  In  this  paper,  we  propose  a  new  evaluation  method 
of  HRV. 

HRV  includes  many  frequency  components,  and  vari¬ 
ous  information  can  be  obtained  from  them  through  the 
frequency  domain  analysis  [1],  [2].  In  the  report  of  Sayers, 
which  is  the  pioneer  research  in  this  area,  three  peaks  exist 
on  the  power  spectrum  of  HRV.  The  lower  frequency  com¬ 
ponent  (0.02  —  0.06[Hz])  is  influenced  by  thermoregulation, 
and  the  middle  frequency  component  (0.07  —  0.14[Hz])  is 
influenced  by  blood  pressure  regulation,  and  the  high  fre¬ 
quency  component  (0.15  —  0.50[Hz])  is  influenced  by  res¬ 
piration  respectively.  However,  if  the  power  spectrum  of 
HRV  is  calculated  using  fast  Fourier  transform,  it  expresses 
rough  information  in  a  fixed  period  of  the  time  series  signal, 
and  the  dynamic  changes  of  the  autonomic  nerve  activity 
cannot  be  expressed.  It  is  difficult  to  analyze  the  non¬ 
stationary  pattern  of  HRV  using  this  method  during  the 


exercise.  To  overcome  this  difficulty,  The  wavelet  trans¬ 
form  (WT),  which  extracts  local  features  of  HRV  in  the 
time-frequency  domain,  is  proposed  [3]. 

The  changes  in  the  spectrum  pattern  from  physical  and 
mental  stress  are  different  among  individuals.  The  sub¬ 
jective  feeling  of  the  subjects  is  also  different.  These  are 
the  problems  of  the  spectrum  analysis  of  HRV.  Most  pre¬ 
vious  studies  defined  specified  frequency  ranges  such  as 
the  low  frequency  component  (LF )  and  the  high  frequency 
component  (HF)  on  the  power  spectrum  of  HRV,  and  ex¬ 
tracted  an  integrated  or  maximum  value  of  the  power  in 
each  range.  However,  this  method  is  not  always  applicable 
because  the  ranges,  scales  and  speeds  of  the  changes  in  the 
spectrum  are  affected  by  various  factors  such  as  individual 
variations,  body  conditions,  circadian  rhythms  and  so  on. 

On  the  other  hand,  some  approaches  using  a  neural  net¬ 
work  have  been  attempted.  These  approaches  have  real¬ 
ized  the  adaptive  signal  processing  of  the  ECG.  Minami  et 
al.  combined  the  feature  extraction  by  Fourier  transform 
and  a  back-propagation  neural  network  (BPN)  [4],  and  de¬ 
tected  the  tachyarrhythmia  in  real-time.  Fahoum  et  al. 
combined  the  WT  and  an  R,BF  neural  network  (RBFN), 
in  order  to  detect  life-threatening  cardiac  arrhythmias  [5]. 
The  purpose  of  these  reports  was  to  detect  the  abnormal 
waveform  on  the  ECG.  The  fact  that  HRV  caused  by  phys¬ 
ical  and  mental  stress,  etc.  was  not  stated.  Also,  there  is  a 
problem  in  that  large  numbers  of  training  data  and  learn¬ 
ing  iterations  are  needed,  if  the  BPN  is  utilized  for  the 
processing  of  a  complicated  signal. 

Authors  have  developed  a  new  statistical  neural  net¬ 
work  called  Log-Linearized  Gaussian  Mixture  Network 
(LLGMN)  [6],  [7].  This  network  is  structured  based  on 
a  Gaussian  mixture  model  and  a  log-linear  model,  and  can 
achieve  higher  performance  in  discriminating  for  electroen¬ 
cephalograms  and  electromyograms  than  other  neural  net¬ 
works.  The  LLGMN  can  learn  the  changes  of  the  signal 
patterns  due  to  the  differences  among  individuals,  differ¬ 
ent  locations  of  the  electrodes,  time  variations  caused  by 
fatigue  or  sweat,  and  so  on.  Authors  also  proposed  the 
Recurrent  Log-Linearized  Gaussian  Mixture  Network  (R- 
LLGMN)  [8].  The  R-LLGMN  uses  the  recurrent  connec¬ 
tion  added  to  the  units  of  the  LLGMN  in  order  to  discrim¬ 
inate  a  time  sequence  of  the  signals  with  high  accuracy. 
The  R-LLGMN  includes  a  hidden  Markov  Model  (HMM) 

[9]  in  its  structure  and  can  modify  the  weight  coefficients 
by  the  back-propagation  through  time  (BPTT)  algorithm 

[10] .  The  weight  coefficients  have  no  statistical  constraints 
like  the  HMM  (e.  g.,  1  >  transition  probability  >  0,  stan¬ 
dard  deviation  >  0),  so  that  the  R-LLGMN  can  realize 
a  higher  learning  ability  than  the  HMM  for  even  a  small 
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Fig.l  Structure  of  tlie  signal  processing. 


number  of  training  data. 

This  paper  proposes  a  new  evaluation  method  of  HRV, 
which  can  evaluate  the  physical  and  mental  stress  based 
on  the  physiological  index.  In  the  proposed  method,  the 
local  changes  of  HRV  in  the  time-frequency  domain  are  ex¬ 
tracted  by  the  WT,  and  they  are  used  for  the  input  data 
of  the  R-LLGMN.  The  R-LLGMN  incorporates  the  HMM 
and  can  evaluate  HRV  patterns,  which  are  affected  by  var¬ 
ious  factors  such  as  individual  variations,  body  conditions, 
circadian  rhythms  and  so  on. 


where  ri;  is  a  scale  parameter  which  selects  the  extracting 
frequency  range,  and  b  is  a  shift  parameter  which  selects 
the  extracting  time  period.  4’{t)  indicates  a  mother  wavelet 
(Gabor  function)  defined  as 

1  t 2 

fP(t)  =  77=  exp[-  -7-  +  iujot\,  (2) 

2y?ra  4a 

where  the  parameter  u>q  is  settled  as  uiq  =  27t/o,/o  =  0.5, 
and  the  parameter  a,  which  regulates  the  time  width  of 
the  Gabor  function,  is  calculated  as 


II  Evaluation  method  of  HRV 

Figure  1  shows  the  structure  of  the  signal  processing, 
which  consists  of  the  measuring  part,  the  feature  extrac¬ 
tion  part  and  the  evaluation  part.  The  measuring  part 
measures  the  HRV  time  series  based  011  the  R-R,  intervals, 
and  the  feature  extraction  part  extracts  the  feature  pat¬ 
terns  of  this  time  series  in  the  time- frequency  domain.  The 
evaluation  part  learns  and  evaluates  the  feature  pattern  us¬ 
ing  the  R-LLGMN.  The  details  of  each  part  are  explained 
in  the  following  sections. 

A  Meat  luring  pari. 

The  EGG  is  monitored  with  1.0 [kHz]  sampling  fre- 
quency  (Polygraph  460,  NEC  San-ei  Instruments,  Ltd.) 
and  the  HRV  time  series  is  sampled  based  on  the  R-R, 
intervals.  Then,  it  is  smoothed  according  to  the  3rd  or¬ 
der  spline  curve  fitting  and  re-sampled  as  /qj(i)[msec]  with 
2.0[Hz]  sampling  frequency,  where  i  indicates  the  i-th  sam¬ 
pled  data. 

B  Feature  extraction  pari 

This  part  extracts  the  feature  patterns  from  hp(i). 
First,  the  mean  values  hpm(i)  (i  >  It)  and  standard  devi¬ 
ations  h.pstdi'i)  are  calculated  every  It  [sec]  as  the  time  do¬ 
main  information.  Then,  multiple  frequency  components 
are  extracted  using  the  WT  as  the  frequency  domain  in¬ 
formation  [3].  Here,  let  us  consider  a  continuous  WT  of 
f(t).  This  transformation  is  defined  as 

(W4.f)(ahb)  =  ^j=f  f(m—)dt ,  (1) 

y/Cll  J  ai 


a  = 


wo  log  2 


(3) 


The  scale  parameter  a;,  in  (1)  is  calculated  as  a;,  = 
a!0  (I  =  1, 2,  ■  *  • ,  L  —  I).  «o  is  defined  as 


«o  =  exp[ 


logu?ma,r  -  E>gw„ 


L  -  1 


(4) 


where  uj,n,a.\-  —  ^0?  rn,jn  —  27r_/),,y;,, ,  —  0.01  are  the 

extracting  maximum  and  minimum  angular  frequencies. 

Using  the  above  equations  (1)  ~  (4),  the  power  of  WT 
|(IE#,/)(rp,i)|  is  calculated.  The  frequency  components, 
which  are  calculated  by  the  scale  parameters  ao  ~  at— 1, 
are  divided  into  S  equal  ranges  and  averaged  within  each 
range.  They  are  filtered  out  through  the  4th  order  but- 
terworth  filter  (cut-off  =  Cf%  and  the  smoothed  signals 
hpw>s(i)  (s  =  1*2,  * "  ,S)  are  extracted. 

Finally,  the  feature  patterns  in  the  time-frequency  do¬ 
main  hp(i)  =  [hpm(i),hpatd(i),h.pWrl(i),hpm>2(i),-  , 
hpw  sin)\ 1  are  normalized  by  the  mean  values  during  the 
rest,  and  re-sampled  as  x(n)  =  [aq(«),. 1^2 (n),a:a («■),-  •  • , 
aq>(n)]T  £  3?-°  every  /„  [sec],  where  n  indicates  the  n-tli 
feature  pattern. 

C  Evaluation  pari 

The  R-LLGMN  [8]  is  used  in  this  part  in  order  to  cope 
with  non-linear  and  nan-stationary  characteristics  of  the 
HRV  patterns  caused  by  individual  variations,  body  con¬ 
ditions,  circadian  rhythms  and  so  on.  The  R-LLGMN  in¬ 
cludes  the  HMM  [9]  in  its  structure,  and  can  realize  higher 
learning  ability  than  the  HMM  for  even  a  small  number 
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of  training  data  because  the  weight  coefficients  of  the  R- 
LLGMN  have  no  constraints  such  as  the  statistical  prop¬ 
erties  of  the  HMM  (e.  g.,  1  >  transition  probability  >  0, 
standard  deviation  >  0).  The  R-LLGMN  receives  the  fea¬ 
ture  patterns  x(n)  and  outputs  the  a  pasteriffri  probability 
for  the  class,  which  corresponds  to  the  state  of  physical  and 
mental  stress. 

Before  evaluation  of  the  input  patterns,  the  R-LLGMN 
must  be  learned.  The  BPTT  algorithm  [10]  is  used  be¬ 
cause  of  the  recurrent  connections.  The  time  history  of 
the  input  pattern  is  considered  from  the  previous  N„  sam¬ 
ples.  Moreover  the  terminal  attractor  [6]  is  incorporated 
with  the  BPTT  in  order  to  regulate  the  convergence  time 
of  the  learning  process. 

Ill  Experiments 

The  experiments  were  carried  out  in  order  to  examine 
the  ability  of  the  proposed  method.  The  mental  workload, 
as  an  example  of  the  physical  and  mental  stress,  was  given 
to  the  subjects,  and  the  subjective  rating  scores  of  the 
mental  stress  were  estimated  using  the  HRV  patterns. 

A  Experimental  conditions 

The  experiments  were  performed  for  five  subjects 
(male/female=4/l,  age  =  31.6  ±  5.5).  They  were  seated 
at  a  desk,  and  a  color  display  (15  inch,  HMD-A101,  Sony 
Corp.)  was  set  at  a  distance  of  60  [cm]  apart  from  their 
eyes.  Integer  numerals  were  displayed  for  2.0[sec]  in  the 
center  of  the  display,  and  the  subjects  were  asked  to  input 
the  same  number  after  fading  out  the  number.  The  font 
size  of  the  displayed  numbers  was  54[point],  and  a  ten- key 
pad  of  the  keyboard  was  used  for  the  input.  During  the 
experiments,  the  EGG  signal  was  measured  based  on  the 
bipolar  derivation  method,  and  there  was  no  indication  of 
the  subject’s  respiration. 

The  subjects  were  asked  to  take  a  rest  for  5.0  inin|. 
Then,  the  sessions  1  ~  9  were  executed.  Each  session 
consisted  of  the  input  task  (2.0[min])  and  the  subjective 
evaluation  (about  30.0  [sec;]).  The  digit  of  displayed  num¬ 
bers  was  equal  to  the  session  number,  and  the  number  was 
displayed  for  2.0[sec].  The  subjective  evaluation  of  the 
mental  stress  was  expressed  in  five  levels,  where  Level  5 
indicated  the  most  stressful  conditions.  Each  subject  car¬ 
ried  out  two  sets  of  this  time  schedule.  The  pattern  which 
was  extracted  in  the  first  set  was  used  as  the  learning  data, 
and  one  pattern  in  the  second  set  was  used  as  the  evalua¬ 
tion  data. 

The  parameters  in  the  feature  extraction  part  were  set¬ 
tled  as  It  =  5.0 [six:].  I„  =  5.0[see],  Of  =  0.005[Hz],  S  =  8, 
D  =  10.  The  scaLe  parameter  of  the  WT,  which  selects  the 
extracting  frequency  range,  was  L  =  160,  where  l  =  159, 
l  =  131,  1  =  0  corresponded  to  0.01  [TTz].  0.25[Hz],  0.5[Hz] 
respectively.  In  the  evaluation  part,  the  number  of  out¬ 
put  units  corresponded  to  the  one  of  the  stress  levels.  The 
number  of  learning  samples  was  N  =  180(20  in  each  ses¬ 
sion).  The  time  history  of  the  input  pattern  was  considered 
from  the  previous  Ns  =  5  samples  in  the  BPTT. 

B  Experimental  results 
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Fig.  2  An  example  of  the  evaluation  results 


Figure  2  shows  an  example  of  the  experimental  result, 
which  shows:  digits  of  the  displayed  numbers  (session  num¬ 
bers),  the  HRV  signal,  the  WT  of  the  HRV  signal,  input 
signaLs  of  the  R-LLGMN,  subjective  rating  scores,  mean 
values  of  the  estimated  scores,  standard  deviations  of  the 
estimated  scores  and  rates  of  correct  answer.  The  WT  of 
the  HRV  signal  is  darkened  as  its  power  increases.  The 
mean  values  and  the  standard  deviations  of  the  estimated 
scores  are  calculated  for  10  kinds  of  initial  weight  coeffi¬ 
cients,  which  are  randomly  chosen. 

It  can  be  seen  that  the  R.-LLGMN  estimated  the  grad¬ 
ual  increase  of  the  mental  stress  successfully,  though  the 
estimated  scores  of  the  R-LLGMN  increased  earlier  than 
the  subjective  rating  scores.  The  standard  deviations  of 
the  estimated  scores  were  considerably  small.  The  corre¬ 
lation  coefficient  between  subjective  rating  scores  and  the 
estimated  scores  was  0.89.  The  estimation  accuracy  of  the 
R-LLGMN  remarkably  decreased  when  digits  of  the  dis¬ 
played  number  was  9.  In  this  case,  the  difficulty  of  the 
task  seemed  to  saturate  the  ability  of  the  subject. 

Next,  the  evaluation  results  for  five  subjects  are  shown 
in  Table  1.  The  results  of  Subject  E  corresponds  to  the  Fig. 
2.  The  table  shows  the  subjective  rating  scores,  estimated 
scores  and  the  rates  of  correct  answers  for  the  sessions 
1  ~  9.  The  mean  values  and  the  standard  deviations  of 
the  estimated  scores  were  calculated  for  10  kinds  of  initial 
weight  coefficients. 


Subjects 

Digits 

1 

2 

3 

4 

5 

6 

7 

8 

9 

A 

Subjective  rating  score 

1 

1 

1 

1 

2 

3 

4 

5 

5 

ESm 

1.27 

1.09 

1.00 

1.16 

2.73 

3.12 

2.95 

4.57 

5.00  Cor  =  0.921 

E  Sstd 

0.44 

0.41 

0.00 

0.53 

0.64 

0.33 

0.32 

1.24 

0.00 

Rate  of  correct  answer 

1.00 

1.00 

1.00 

1.00 

0.95 

0.89 

0.67 

0.40 

0.00 

B 

Subjective  rating  score 

1 

2 

3 

3 

3 

4 

4 

5 

5 

ESm 

1.23 

1.65 

2.95 

3.00 

2.25 

3.79 

4.28 

5.00 

5.00  Cor  =  0.920 

ESstd 

0.51 

0.48 

0.21 

0.00 

1.79 

0.72 

0.57 

0.00 

0.00 

Rate  of  correct  answer 

1.00 

1.00 

1.00 

1.00 

0.96 

0.84 

0.78 

0.86 

0.26 

C 

Subjective  rating  score 

1 

2 

2 

2 

3 

4 

5 

5 

5 

ESm 

1.00 

2.00 

2.00 

2.00 

3.53 

3.00 

4.73 

5.00 

5.00  Cor  =  0.890 

E  Sstd 

0.00 

0.00 

0.00 

0.00 

0.89 

1.42 

0.68 

0.00 

0.00 

Rate  of  correct  answer 

1.00 

1.00 

1.00 

1.00 

0.96 

0.82 

0.45 

0.29 

0.04 

D 

Subjective  rating  score 

1 

1 

2 

3 

4 

5 

5 

5 

5 

ESm 

1.55 

1.01 

1.00 

3.49 

4.38 

5.00 

5.00 

5.00 

3.41  Cor  =  0.855 

E  Sstd 

0.89 

0.12 

0.00 

0.49 

0.49 

0.00 

0.00 

0.00 

0.81 

Rate  of  correct  answer 

1.00 

1.00 

1.00 

0.93 

0.78 

0.40 

0.19 

0.07 

0.00 

E 

Subjective  rating  score 

1 

1 

1 

2 

3 

4 

5 

5 

5 

ESm 

1.00 

1.00 

1.00 

2.74 

3.93 

5.00 

5.00 

4.97 

3.00  Cor  =  0.888 

ESstd 

0.00 

0.00 

0.00 

1.12 

0.25 

0.00 

0.00 

0.16 

0.00 

Rate  of  correct  answer 

1.00 

1.00 

1.00 

1.00 

1.00 

0.86 

0.74 

0.19 

0.00 

ESm  :  Mean  values  of  the  estimated  score 
ESstd  '■  Standard  deviations  of  the  estimated  score 

Cor  :  Correlation  coefficient  between  subjective  rating  score  and  estimated  score 

Table  1  Experimental  results  for  five  subjects. 


We  see  from  the  table  that  the  changes  in  the  subjective 
rating  score  and  the  rate  of  correct  answer  were  different 
among  individuals.  Under  such  situations,  the  R-LLGMN 
estimated  the  gradual  increase  of  the  mental  stress  suc¬ 
cessfully.  The  correlation  coefficients  between  subjective 
rating  scores  and  the  estimated  scores  were  0.92  ~  0.89. 
The  evaluation  performance  of  Subject  D  decreased  like 
Fig.  2  (Subject  E)  during  Session  9. 

IV  Conclusion 

This  paper  proposed  a  new  evaluation  method  of  heart 
rate  variability  in  order  to  evaluate  the  physical  and  men¬ 
tal  stress  based  on  the  physiological  index.  In  this  method, 
the  feature  patterns  of  HKV  were  extracted  by  using  the 
WT,  and  the  R-LLGMN  learned  and  evaluated  these  pat¬ 
terns.  In  the  experiments,  the  subjective  rating  scores  of 
the  subject’s  mental  stress  were  evaluated  with  high  accu¬ 
racy. 

In  the  future,  we  would  like  to  develop  a  monitoring 
system,  which  incorporates  the  proposed  method,  in  order 
to  evaluate  HRV  patterns  in  daily  life. 
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